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1  Summary 

We  have  made  progress  in  the  following  areas 

1.  The  Loading  Problem:  The  proof  that  the  loading  problem  for  analog  neural  networks  with 
only  6  nodes  is  A/’T’-complete,  first  announced  by  the  author  in  1992,  was  completed  early  in 
1993. 

2.  Threshold  Functions  with  Real-  Valued  Inputs:  Foundational  results  on  linear  threshold  func¬ 
tions  with  Boolean  inputs  have  been  extended  to  real-valued  inputs. 

3.  Dynamic  Range  of  Weights:  Hastad’s  lower  bound  on  the  dynamic  range  of  weights  for  linear 
threshold  functions  has  been  improved  slightly. 

4.  Hopfield  Networks  for  Knight’s  Tours:  Takefuji’s  a  Hopfield  network  for  knight’s  tours  was 
analyzed  theoretically  and  experimentally  in  comparison  with  conventional  algorithms.  We 
have  proven  a  new  lower  bound  for  the  number  of  knight’s  tours  on  a  generalized  chessboard, 
which  throws  further  light  on  why  Takefuji’s  Hopfield  network  for  knight’s  tours  works  as  well 
as  it  does. 

5.  V CS -completeness  of  the  stable  state  problem  for  Hopfield  nets:  It  has  been  proved  that  the 
problem  of  finding  stable  states  for  some  restricted  classes  of  Hopfield  nets  is  just  as  difficult 
as  the  general  problem. 

6.  Exponential  lower  bounds  for  planar  Hopfield  networks:  We  have  constructed  planar  Hopfield 
networks  that  take  time  2n/3  to  converge  under  sequential  operation,  and  2n/7  to  converge  in 
parallel  operation. 

‘Author’s  address:  Department  of  Computer  Sciences,  University  of  North  Texas,  P.O.  Box  13886,  Denton,  TX 
76203-6886,  U.S.A.  Electronic  mail:  ian9cs.unt.edu. 
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2  Statement  of  Work 


The  Principal  Investigator  and  his  Research  Assistant  shall  conduct  research  into  the  scalability  of 
neural  networks.  The  research  shall  analyze  the  resource  requirements  of  neural  network  compu¬ 
tation  and  learning  as  a  function  of  the  size  of  the  input  domain.  The  project  will  investigate  the 
general  topic  of  scalability  in  neural  networks,  including  the  specific  areas  of  research  listed  below 
and  any  related  subjects  whose  interest  and  relevance  are  uncovered  in  the  course  of  this  research. 

1.  The  researchers  shall  investigate  whether  the  loading  problem  for  analog  neural  networks  with 
a  small  fixed  number  of  nodes  is  A/T’-complete. 

2.  A  hitherto  unexplored  complexity  class,  called  ATC°,  shall  be  investigated.  This  class  consists 
of  the  problems  that  can  be  solved  by  a  neural  network  with  a  fixed  number  of  layers  and 
a  polynomial  number  of  nodes,  in  which  the  first  layer  of  nodes  compute  linear  threshold 
functions  and  nodes  in  the  remaining  layers  compute  conjunction  and  disjunction.  Open 
questions  about  the  relationships  between  AT C°  and  the  well-studied  complexity  classes  AC0 
and  TC°  shall  be  explored. 

3.  The  problem  of  determining  improved  upper  and  lower  bounds  on  the  size  of  weights  in  neural 
networks  shall  be  investigated. 

4.  Open  problems  concerning  the  running  time  of  Hopfield  networks  used  to  solve  local  opti¬ 
mization  problems  shall  be  investigated.  The  theory  of  VCS  completeness  shall  be  used  to  to 
determine  whether  Hopfield  networks  for  local  optimization  problems  shall  require  running 
time  that  is  exponential  in  the  number  of  inputs. 

3  Status  of  Research 

Progress  has  been  made  in  six  areas:  the  complexity  of  the  loading  problem,  properties  of  threshold 
functions  with  real- valued  inputs,  lower  bounds  for  the  dynamic  range  of  weights  in  linear  threshold 
functions,  Hopfield  networks  for  knight’s  tours,  Hopfield  networks  for  local  optimization,  and  lower 
bounds  for  Hopfield  network  convergence.  Each  is  discussed  below  in  a  separate  subsection.  • 

3.1  The  Loading  Problem 

Judd  [10,  11,  12,  13]  has  shown  that  the  problem  of  loading  simple  tasks  onto  neural  networks  with 
a  fixed  architecture  is  AfP-complete,  which  implies  that  there  is  quite  likely  to  be  no  fast  general- 
purpose  learning  algorithms  even  for  quite  simple  architectures  and  node  function  sets.  Surprisingly, 
it  was  shown  by  Blum  and  Rivest  [1,  2]  that  the  loading  problem  is  .A/P-complete  even  for  networks 
consisting  of  only  3  nodes  when  the  node  function  set  is  the  set  of  linear  threshold  functions.  One 
limitation  of  the  result  of  Blum  and  Rivest  is  that  it  holds  only  for  node  function  sets  that  are 
exactly  linear  threshold  functions.  Judd’s  techniques  work  for  any  node  function  set  that  includes 
ACi,  and  hence  apply  to  analog  neural  networks,  but  he  has  no  results  for  a  fixed  number  of  nodes. 
Our  new  result  is  as  general  as  Judd’s,  but  has  a  fixed  node  bound.  Specifically,  we  show  that 
the  loading  problem  for  a  6-node  neural  network  with  node  function  set  AC\  is  A/’P-complete,  and 
deduce  deduce  that  the  loading  problem  for  a  6-node  analog  neural  network  is  A/T-hard. 

How  does  one  interpret  the  meaning  of  vV'P-completeness  results  for  the  loading  problem?  They 
imply  that  any  learning  algorithm  that  takes  as  input  a  set  of  tasks  and  a  fixed  architecture  runs  the 
risk  of  taking  exponential  time  in  the  worst  case  even  for  architectures  drawn  from  quite  innocuous 
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architecture  classes.  This  can  be  avoided  either  by  limiting  the  node  function  set  and  choosing 
specific  architectures  for  which  the  loading  problem  is  not  intractable,  by  allowing  the  architecture 
to  change  during  learning,  or  by  only  loading  task  sets  that  do  not  cunningly  encode  yV'P-complete 
problems.  Results  such  as  ours  indicate  that  even  very  simple  architectures  and  node  function  sets 
can  have  task  sets  that  encode  VP-complete  problems.  This  can  be  a  major  pitfall  for  the  unwary 
neural  network  designer,  and  may  explain  why  some  learning  algorithms  scale  badly. 

3.2  Threshold  Functions  with  Real- Valued  Inputs 

Discrete  linear  threshold  functions  are  popular  as  node  functions  in  discrete  neural  networks.  While 
there  is  a  fairly  substantial  body  of  foundational  results  on  linear  threshold  functions  with  Boolean 
inputs,  the  corresponding  body  of  work  for  linear  threshold  functions  with  real  inputs  is  not  so 
well-studied.  We  have  shown  that  over  the  real  domain,  all  weight-sets  of  a  given  linear  threshold 
function  are  a  real  multiple  of  each  other.  In  contrast,  it  is  also  shown  that  every  separable  linear 
threshold  function  over  an  infinite  but  bounded  domain  has  an  integer  weight-set.  The  latter  result 
is  applied  to  extend  the  upper  bound  on  the  weights  in  an  integer  weight-set,  and  the  running  time 
and  mistake  bounds  of  the  perceptron  learning  algorithm  from  the  Boolean  domain  to  all  finite  real 
domains. 

3.3  Dynamic  Range  of  Weights 

Hastad  [17]  has  shown  that  there  exist  linear  threshold  functions  that  requires  weights  approxi¬ 
mately  nn!1 2 3 4~n.  We  have,  by  a  careful  analysis,  tightened  the  lower  bound  to  the  following: 

n(n— logn— l)/2e7(log2n+31ogn+8)(log2n  _  logn  +  ^ 

g87n^2n— 1 

where  0  =  log(3/2)  «  0.585,  7  =  ln(4/3)  «  0.288. 

3.4  Hopfield  Networks  for  Knight’s  Tours 

We  began  our  study  of  the  running  time  required  by  Hopfield  nets  for  solving  optimization  problems 
by  embarking  on  experimental  work.  A  knight’s  tour  is  a  series  of  moves  made  by  a  knight  visiting 
every  square  of  an  n  x  n  chessboard  exactly  once.  The  knight’s  tour  problem  is  the  problem  of 
constructing  such  a  tour,  given  n.  The  knight’s  tour  problem  is  interesting  because  it  is  a  restricted 
case  of  the  travelling  salesperson  problem.  Takefuji  and  Lee  [19]  (reproduced  apparently  verbatim  in 
Takefuji  [18,  Chapter  7])  recently  proposed  a  Hopfield-style  network  for  the  knight’s  tour  problem. 
We  have  compared  their  algorithm  with  existing  algorithms,  both  sequentially  and  in  parallel,  both 
for  producing  single  tours  and  producing  multiple  tours.  We  observed  that: 

1.  The  Hopfield  network  was  too  slow  to  use  for  boards  larger  than  26  x  26. 

2.  A  random  walk  algorithm  that  combines  two  algorithms  invented  by  Euler  (1759)  and  Warns- 
dorf  (1800s)  was  tested.  It  was  found  to  be  practical  for  boards  up  to  66  x  66. 

3.  A  divide-and-conquer  algorithm  quickly  gave  solutions  for  boards  of  size  up  to  1000  x  1000. 

4.  Comparative  running  times  can  be  seen  in  Figure  1. 

We  can  conclude  that: 

1.  Experimental  evidence  has  shown  that  both  the  Hopfield  net  and  the  random  walk  algorithm 
run  in  exponential  time.  Even  so,  the  random  walk  algorithm  is  vastly  superior. 
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2.  Theoretical  analysis  has  shown  that  the  divide-and-conquer  algorithm  runs  in  linear  time. 

Hence  the  divide-and-conquer  algorithm  is  far  preferable  to  the  other  two. 

3.  In  this  instance,  neural  networks  don’t  scale  as  well  as  conventional  algorithms. 

Parallel  speedup  of  the  Hopfield  network  cannot  reduce  the  exponential  running  time  signifi¬ 
cantly  unless  exponentially  many  processors  are  used.  In  contrast,  the  divide-and-conquer  algorithm 
can  be  implemented  in  optimal  time  (that  is,  proportional  to  the  diameter  of  the  interconnection 
pattern)  using  polynomially  many  processors  on  many  popular  architectures.  This  includes: 

•  time  0(n2/p)  by  a  bounded  degree  network  with  p  processors  for  all  p  =  0(n2/  log  n), 

•  time  0(n2/p2)  by  a  p  x  p  mesh  for  all  p  <  n2/3, 

•  0(1)  time  on  an  n  x  n  mesh  with  multiple  CREW  buses,  and 

•  0(1)  time  on  a  CREW  PRAM  with  0(n2)  processors. 

Similar  research  has  shown  that  the  divide-and-conquer  algorithm  is  much  better  than  the 
Hopfield  network  and  the  random*  walk  algorithm  at  finding  large  numbers  of  tours.  We  were  able 
to  find  a  lower  bound  of  0(1.1259"  )  on  the  number  of  knight’s  tours  on  an  n  x  n  board  for  all  even 
n  >  6,  and  a  stronger  lower  bound  of  0(1.2484"  )  when  n  is  of  the  form  6x2*  for  some  k  €  IM. 

Neural  networks  research  has  progressed  to  the  point  at  which  the  mere  fact  of  existence  of  a 
neural  network  solution  for  a  problem  is  no  longer  interesting.  Long  term  interest  in  neural  network 
research  can  only  be  sustained  if  neural  networks  can  compete  with  conventional  algorithm  design 
techniques.  A  new  algorithm  is  significant  if  it  can  be  demonstrated  beyond  reasonable  doubt  to 
be  more  practical,  more  effective,  or  more  efficient  than  existing  techniques.  We  have  found  that 
this  is  not  the  case  for  a  neural  network  for  the  knight’s  tour  problem,  neither  for  finding  single 
tours  sequentially  or  in  parallel,  nor  for  finding  bounds  on  the  number  of  tours. 

We  have  worked  with  Olaf  Kyek  and  Ingo  Wegener  at  the  University  of  Dortmund  to  show  that 
for  all  even  n  >  12,  the  number  of  knight’s  tours  on  an  n  x  n  chessboard  is 

•  at  least  1.1646"2, 

•  at  most  4"2, 

•  asymptotically  Q(1.3535"2). 

This  indicates  that  knight’s  tours  are  reasonably  dense,  but  decrease  exponentially  as  n  approaches 
infinity.  This  helps  explains  why  the  performance  of  Takefuji’s  neural  network  decreases  so  rapidly 
as  n  increases. 

3.5  Hopfield  Nets  for  Local  Optimization 

P£<?-completeness  is  the  analog  of  Ar'P-completeness  for  local  optimization  problems.  A  local  op¬ 
timization  problem  that  is  PZ^S-complete  is  considered  very  unlikely  to  have  a  polynomial  time 
algorithm.  P£<S-completeness  was  invented  by  Johnson,  Papadimitriou,  and  Yannakakis  [9].  Re¬ 
cently,  Papadimitriou,  Schaffer,  and  Yannakakis  [16]  showed  that  the  problem  of  finding  stable 
states  in  a  Hopfield  network  in  P£5-complete.  This  is  both  good  news  and  bad  news  for  propo¬ 
nents  of  Hopfield  nets.  It  shows  that  they  are  powerful  computational  models  that  are  capable 
of  encoding  problems  that  are  difficult  for  conventional  computers,  but  also  shows  that  they  are 
unlikely  to  run  in  polynomial  time. 

The  argument  of  Papadimitriou,  Schaffer,  and  Yannakakis  [16]  shows  that  the  stable  state 
problem  for  Hopfield  nets  is  "P£<S-complete  in  general,  but  does  not  address  the  case  of  restricted 
Hopfield  nets.  The  Hopfield  nets  found  in  the  literature  typically  have  restrictions  on  the  underlying 
graph.  We  have  preliminary  proofs  of  the  following  results.  The  stable  state  problem  for  Hopfield 
nets  is  still  'P£5-complete  even  for: 
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Figure  1:  Running  times  for  the  Hopfield  network,  random  walk  algorithm,  and  the  divide-and- 
conquer  algorithm  on  an  n  x  n  board.  Note  that  the  time  scale  is  logarithmic. 


Figure  2:  A  grid  graph. 


1.  Hopfield  nets  whose  interconnection  pattern  is  bipartite,  that  is,  the  neurons  are  divided  into 
two  sets  L  and  R  such  that  if  there  is  a  connection  between  neurons  v  and  w,  then  v  €  L  and 
w  G  R,  or  vice-versa. 

2.  Hopfield  nets  of  degree  3,  that  is,  every  neuron  is  connected  to  at  most  3  other  neurons. 

3.  Hopfield  nets  whose  interconnection  pattern  is  a  very  simple  grid  graph  with  crossovers  in 
each  grid  element  (see  Figure  2). 

4.  Hopfield  nets  whose  interconnection  pattern  is  one  of  a  class  of  graphs  used  heavily  in  par¬ 
allel  computation,  including  the  hypercube,  the  butterfly,  the  cube-connected  cycles,  and 
the  shuffle-exchange  (see,  for  example,  Leighton  [14]).  Hopfield  nets  whose  interconnection 
pattern  is  the  dual  of  the  knight’s  graph,  used  by  Takefuji  and  Lee  [19]. 

Result  (1)  is  proved  by  adapting  a  technique  due  to  Bruck  and  Goodman  [4,  5].  Results  (2-4) 
are  proved  using  a  graph  embedding  technique.  It  is  first  shown  that  the  stable  state  problem  for 
Hopfield  nets  with  interconnection  pattern  from  an  arbitrary  class  of  graphs  C  is  P£<S-complete  if 
an  arbitrary  graph  G  can  be  embedded  in  a  graph  from  C  with  congestion  1,  load  1,  and  polynomial 
expansion.  (An  embedding  of  a  graph  G\  =  (Vi,Ei)  into  Gi  =  (V^,  E^)  is  a  function  / :  V]  -*  V2 
such  that  for  all  ( v,w )  €  Ei,  there  exists  a  path  between  f(v)  and  f(w)  in  E%.  The  congestion  is 
the  maximum  number  of  such  paths  that  uses  any  single  edge,  the  load  is  the  maximum  number  of 
vertices  in  V\  that  have  the  same  image  under  /,  and  the  expansion  is  the  size  of  V2  as  a  function 
of  the  size  of  V\.)  We  are  then  able  to  show  that  suitable  embeddings  exist  for  the  required  classes 
of  graphs. 

^^-completeness  results  for  Hopfield  nets  can  be  interpreted  in  two  ways.  An  optimistic 
interpretation  is  that  they  identify  the  problem  of  finding  stable  states  in  a  Hopfield  network  as  a 
key  problem  in  the  theory  of  local  optimization.  A  pessimistic  interpretation  is  that  it  is  unlikely 
that  a  fast  algorithm  can  be  found  for  finding  stable  states  in  Hopfield  networks,  since  such  an 
algorithm  would  provide  fast  algorithms  for  problems  that  have  resisted  such  attempts  for  a  long 
period  of  time.  We  have  provided  strong  evidence  that  there  are  no  polynomial  time  algorithms  for 
finding  stable  states  in  some  very  simple  classes  of  Hopfield  nets,  and  in  particular  that  each  class 
includes  Hopfield  networks  that  do  not  converge  in  polynomial  time. 

3.6  Lower  Bounds  for  Hopfield  Network  Convergence 

A  planar  Hopfield  network  is  one  whose  interconnection  graph  is  planar,  that  is,  can  be  drawn 
on  the  Euclidean  plane  without  crossing  edges.  Haken  and  Luby  [8]  describe  a  planar  Hopfield 
network  that  provably  takes  exponential  time  to  converge.  Yet  there  exists  an  algorithm  for  the 
stable  state  problem  that  runs  in  polynomial  time  on  all  Hopfield  networks.  The  proof  follows  from 
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the  fact  that  the  maximal  cut  in  a  planar  graph  can  be  found  in  polynomial  time  (see,  for  example, 
Hadlock  [7]),  combined  with  results  of  Papadimitriou,  Schaffer,  and  Yannakakis  [16].  Thus,  we 
have  proved  that  conventional  computers  are  exponentially  faster  than  planar  Hopfield  networks  in 
sequential  mode. 

We  further  considered  the  question  of  how  long  the  convergence  time  for  a  planar  Hopfield 
network  can  be,  in  both  sequential  and  parallel  operation.  In  the  case  of  sequential  updates,  Haken 
and  Luby  [8]  construct  a  planar  Hopfield  network  on  n  nodes  that  takes  time  2n/7  to  converge.  Goles 
and  Martinez  [6],  building  on  the  work  of  Orponen  [15],  extended  this  to  2”/6  in  the  nonplanar 
case.  Using  different  techniques,  we  were  able  to  prove  a  lower  bound  of  2n/3  in  the  planar  case, 
thus  improving  substantially  on  both  results. 

In  the  case  of  parallel  updates,  Goles  and  Martinez  [6],  and  Bruck  [3]  separately  described  a 
Hopfield  network  that  takes  time  2n/3  to  converge.  There  were  no  corresponding  results  for  planar 
Hopfield  networks  in  parallel  mode.  We  have  constructed  a  planar  Hopfield  network  that  takes 
time  2n/7  to  converge. 
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Texas,  May  1994. 
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Vol.  12,  pp.  19-34,  1996. 
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December  1995  to  Discrete  Applied  Mathematics. 
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January  1996  to  Discrete  Applied  Mathematics. 
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